DISPLAY AND POINTER MANIPULATION USING RELATIVE 

MOVEMENT TO A DEVICE 

REFERENCE TO RELATED DOCUMENTS 
This application is related to the subject matter described in the following two 
5 U.S. Patents: 5,578,813 filed 2 March 1995, issued 26 November 1996 titled 

FREEHAND IMAGE SCANNING DEVICE WHICH COMPENSATES FOR NON- 
LINEAR MOVEMENT; and 5,644,139, filed 14 August 1996, issued 1 July 1997 
titled NAVIGATION FOR DETECTING MOVEMENT OF NAVIGATION 
SENSORS RELATIVE TO AN OBJECT. Both of these Patents have the same 

10 inventors: Ross R. Allen, David Beard, Mark T. Smith and Barclay J. Tullis, and both 
Patents were, at the time that the invention that is the subject of this application was 
made, owned by the same entity or subject to an obligation of assignment to the same 
entity. This application is also related to the subject matter described in U.S. Patent 
5,786,804 issued 20 July 1998, titled METHOD AND SYSTEM FOR TRACKING 

15 ATTITUDE, and pending divisional application based on the aforementioned issued 
patent serial number 09/022097, and U.S. Patent application 09/052046, HP docket 
number 10980359-1 filed 30 March 1998, titled SEEING EYE MOUSE FOR A 
COMPUTER SYSTEM, all of these were, at the time that the invention that is the 
subject of this application was made, owned by the same entity or subject to an 
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obligation of assignment to the same entity. The three patent documents describe 
techniques of tracking position movement and computer pointing devices. Those 
techniques and devices are components in the preferred embodiment described below. 
Accordingly, U.S. Patents 5,578,813, 5,644,139, and 5,786,804 are hereby 
5 incorporated herein by reference. The related application serial number 09/022097 
describes a computer mouse based on those techniques, therefore U.S. Patent 
application serial number 09/022097, HP docket number 10980359-1 filed 30 March 
1998, titled SEEING EYE MOUSE FOR A COMPUTER SYSTEM is hereby 
incorporated herein by reference. 

10 FIELD OF THE INVENTION 

This invention relates generally to controlling a cursor, or scrolling, on a 
display screen. This invention relates even more particularly to methods and 
apparatus for controlling a cursor or scrolling on small handheld devices. 



15 BACKGROUND OF THE INVENTION 

The use of a hand operated pointing device for use with a computer or other 
electronic device and its display has become almost universal. By far the most popular 
of the various devices is the mouse. More recently, portable computers and personal 
digital appliances (PDA's) have become available that are small enough to be 

20 handheld. Because a mouse is typically designed to fit comfortable in a human hand, 
mice are relatively large when compared to the size of these handheld devices. 
Accordingly, these handheld sized devices may use a pen-like stylus, touch screen, 
buttons, and other things to manipulate a cursor, or control scrolling functions. One 
example of this is the HP CapShare 920 portable e-copier available from Hewlett- 

25 Packard, Co., Palo Alto, California. This device is a handheld scanner that includes a 
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small liquid-crystal display (LCD) screen for displaying menus, dialog boxes, and 
previews of the scanned image. The user interfaces with these menus, etc. by pressing 
various buttons located on the device. For example, to scroll the preview of the 
scanned image to the left, the user presses a button shaped like a leftward pointing 
5 arrow. Another example of a handheld sized device is the HP Jornada 420 palm-sized 
PC available from Hewlett-Packard, Co., Palo Alto, California. This device has an 
LCD screen and five buttons for powering on or off, recording, scrolling though and 
selecting text and pulling up a START menu. 

Unfortunately, multiple buttons with multiple functions, and styluses, lack 
10 some of the advantages that a mouse provides. For example, the graphical user 
^ j interfaces (GUI's) associated with some of the most popular operating systems are 

lu easiest to learn and operate with a moving cursor controlled by a mouse (or trackball, 

[ *3 etc.) . A typical function performed with these systems is the selection of a single 

!!* icon from a distributed field of icons. With multiple buttons, this function may 

n 15 require multiple button presses that select successive icons in orthogonal directions. 

s 5 When the desired icon is finally selected, yet another button must be pressed to 

activate the desired icon and perform some operation. In contrast, a free-floating 
mouse-type and cursor system only requires a natural sliding motion performed with 
one hand to move the cursor and a single button press on the mouse to perform the 
20 operation. Another function performed with these systems is scrolling, panning, and 
navigating to view portions of a document that are not currently shown. With 
multiple buttons, each scroll, pan, or navigation is often done in discrete, 
predetermined, orthogonal, intervals. A free-floating mouse-type system can provide 
smoother movements in non-orthogonal directions that are more natural to use. 
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Accordingly, there is a need in the art for an improved pointing interface 
device and method that is suitable for use with handheld devices. Such an interface 
should provide the intuitive feel and familiarity of a mouse-based system without 
increasing the size or disrupting the unity of package of a handheld device. Such an 
5 interface should also provide an intuitive way of scrolling, panning, or performing 
other navigation functions. Finally, the interface should try to minimize the number 
of buttons on a handheld device because of cost and usability concerns. 



10 SUMMARY OF THE INVENTION 

The invention provides mouse-like functions of smooth, non-orthogonal 
movement of cursors, scrolling, panning, and navigating on a handheld device without 
a detached or semi-detached mouse-like or stylus-like part. By providing mouse-like 
functions, the invention preserves the intuitive feel and familiarity of a mouse based 

15 systems on a handheld device. The invention may be implemented with no moving 
parts, and a minimum number of buttons to perform the functions of the buttons on a 
mouse. Finally, the invention allows existing graphical user interfaces to be used on 
handheld devices without redesigning the workflow or defining special keys for 
navigating a cursor or selecting graphical objects. 

20 An embodiment of the invention uses navigation sensors on a handheld device 

with a display to detect and measure motion of that device relative to a surface in 
close proximity to the navigation sensor. This relative motion may be created by 
moving the entire device around a surface on which the device is place, or it may be 
created by moving a finger or other body part in front of the navigation sensor. The 

25 relative motion of the device is then used to move a cursor on the display, or pan or 
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otherwise navigate what is shown on the display. In one embodiment, the navigation 
sensors are placed on the opposite side of the device as the display. This allows the 
user to slide the device around the surface it is placed on to manipulate the display 
while viewing the face-up display. In another embodiment, the navigation sensors are 
5 placed on the bottom of a two part hinged device. This allows the user to slide the 
device around the surface it is placed on to manipulate the display that is conveniently 
angled for viewing by the hinge. In another embodiment, the navigation sensors serve 
a dual function. The first function is to aid in the capture of a scanned image by a 
handheld scanner. Then, after an image has been acquired, the same navigation 
10 sensors are used to manipulate the display. 

Other aspects and advantages of the present invention will become apparent 
from the following detailed description, taken in conjunction with the accompanying 
drawings, illustrating by way of example the principles of the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 
15 FIG. 1 is an illustration of a handheld digital appliance with attached display. 

FIG. 2A is an illustration of a two part hinged device with attached display in 
an open position. 

FIG. 2B is an illustration of a two part hinged device with attached display in a 
closed position. 

20 FIG. 3 is an illustration of a handheld scanning device with attached display. 

FIG. 4 is an illustration of a handheld scanning device with an attached display 
located on the top of the device. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
FIGS. 1-4 show handheld electronic devices useful for performing any of 
25 multiple computing, business, entertainment, and communication tasks. In addition, 
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FIG. 3 and FIG. 4 show electronic devices that are also useful for scanning an image- 
bearing surface. In FIG.l, the electronic device 1000 includes a display 1002 and two 
buttons 1004, 1006 located on a top surface 1010. Located on a bottom surface is a 
navigation sensor 1008. Navigation sensor 1008 is used to track the movement of the 
5 device 1000 relative to a surface upon which the device is placed. Navigation sensors 
of several types are disclosed and described in the documents mentioned above in the 
section REFERENCE TO RELATED DOCUMENTS. Accordingly, those documents 
mentioned above in that section are hereby incorporated herein by reference. 

Navigation sensor 1008 effectively observes a moving image of the surface in 
£3 io front of navigation sensor 1008. This motion may be created by placing device 1000 

"~ upon a surface and moving the entire device, or by moving a part of the user in front 

'rfi of navigation sensor 1008. Navigation sensor 1008 produces an indication of the 

□ displacement in two planar dimensions between successive observations. This 

l=* indication of displacement is used by device 1000 to manipulate what is shown on 

m 

I s * 15 display 1002. For example, the indications of displacement may be used by device 

! f 1000 to control a cursor that is shown on display 1002. In this manner, the user may 

move device 1000 which would, in turn, move the position of a cursor shown on 
display 1002. This coordination of device movement to cursor movement could then 
be used with a graphical user interface (GUI) and buttons 1004 and 1008 to control 
20 the operation of device 1000 much like the movement of a mouse controls the 
operation of a GUI on a desktop personal computer. 

The indications of displacement may also be used to perform other functions 
that manipulate what is shown on display 1002. For example, the indications of 
displacement may be used to control panning, scrolling, or other image navigation 
25 functions. This would allow the user to move the device 1000 in a desired direction 
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to see regions of an image that are not presently displayed on display 1002. This 
provides the user a natural feel for these operations that is similar to moving a 
magnifying glass over an area of interest. 

In FIG. 2A and 2B, the electronic device 2000 includes a system unit 2002 and 
5 a display unit 2004. The system unit 2002 includes a keyboard 2006, buttons 2014, 
2016, and navigation sensor 2020. Navigation sensor 2020 shown located on the 
bottom surface of system unit 2002. Keyboard 2006 and buttons 2014, 2016 are 
shown located on the top surface of system unit 2002. The display unit 2004 includes 
display panel 2008 and display panel housing 2010. The display unit 2004 is attached 

10 to the system unit 2002 and rotates on hinge 2012 between an open position (shown in 
FIG. 2A) and a closed position (shown in FIG. 2B). Navigation sensor 2020 
effectively observes a moving image of the surface in front of navigation sensor 2020. 
This motion may be created by placing device 2000 upon a surface and moving the 
entire device, or by moving a part of the user in front of navigation sensor 2020. This 

15 relative movement can be used to manipulate what is shown on the display 2008 and 
control the device 2000 as described above. 

FIG. 2A and 2B show buttons 2014 and 2016 located on the top surface of 
system unit 2002. This would allow the user to grasp the system unit between the 
thumb and middle finger and use the index finger to press buttons 2014 and 2016. 

20 However, this is just one of many possibilities. Buttons 2014 and 2016 may be 
located in other positions and other surfaces or may be not included in the device 
completely. For example, if buttons 2014 and 2016 are not included, then the user 
may perform the same functions by pressing certain keys on keyboard 2006 with an 
index finger. Another possibility is to locate buttons 2014 and 2016 on the front 
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vertical surface of system unit 2002. This would allow the user to grasp the unit 
between the thumb and index finger and press these buttons with a thumb. 

FIG. 3 is an illustration of a handheld scanning device 3000 with attached 
display 3002. The scanning device 3000 includes a display 3002 that may provide 
5 almost immediate viewing of a captured image. Scanning device 3000 also includes 
buttons 3004 and 3006. Buttons 3004 and 3008 can be used to aid in interfacing with 
a GUI, image capture, or image navigation. 

In FIG. 3, the forward side of the scanning device 3000 includes navigation 
sensors 3008 and 3010 and imaging sensor array 3012. Typically, when the device is 

10 being used to scan an image, this forward side is placed toward the image to be 

captured and the device is moved around the image to capture it. In this position, the 
navigation sensors 3008 and 3010 are in close proximity to the surface with the image 
on it and are able to effectively observe a moving image of the surface device 3000 is 
placed upon and produces an indication of the displacement in two planar dimensions 

15 between successive observations. Also, display 3002 is angled slightly off 

perpendicular with this forward side to improve the viewability of display 3002 when 
device 3000 is in this position. 

Device 3000 uses navigation sensors 3008 and 3010 to manipulate a GUI or 
scroll through an image while viewing display 3002. Navigation sensors 3008 and 

20 3010 are also used during image capture. This allows the function of one or more of 
the navigation sensors 3008 and 3010 to be shared between display manipulation and 
image capture without additional hardware. It also allows capture verification to be 
very intuitive. To verify the image capture, the user would place the forward side in 
FIG. 3, against a surface and move it around. Alternatively, the user could hold the 

25 device and move a finger or other body part over at least one of the navigation sensors 



HP docket #10991692 



8 



<3 




3008 and 3010. The display 3002 would appear to the user to be moving around the 
captured image. One of buttons 3004 and 3006 could be used to activate a navigate 
mode that moves a captured image around the display in response to the movement of 
device 3000 or movement of the users finger. Finally, since most GUI and scrolling 
5 functions do not require the positional accuracy that scanning does, the indications of 
displacement produced by navigation sensors 3008 or 3010 may be downsampled. 
This would reduce the compute requirements associated with handling the indications 
of displacement when an image was not being captured. 

FIG. 4 is an illustration of a handheld scanning device 4000 with attached 

10 display 4002 on the top of the device. The scanning device 4000 includes a display 
4002 on the top of the device that may provide almost immediate viewing of a 
captured image. Scanning device 4000 also includes buttons 4004, 4006, and 4014 on 
top of the device. Buttons 4016 and 4018 are shown positioned on a vertical side of 
scanning device 4000. Any of buttons 4004, 4006, 4014, 4016, and 4018 can be used 

15 to aid in interfacing with a GUI, image capture, or image navigation. 

In FIG. 4, the botton side of the scanning device 4000 includes navigation 
sensors 4008 and 4010 and imaging sensor array 4012. Typically, when the device is 
being used to scan an image, this bottom side is placed toward the image to be 
captured and the device is moved around the image to capture it. In this position, the 

20 navigation sensors 4008 and 4010 are in close proximity to the surface with the image 
on it and are able to effectively observe a moving image of the surface device 4000 is 
placed upon and produces an indication of the displacement in two planar dimensions 
between successive observations. Also, display 4002 may angled slightly off level to 
improve the viewability of display 4002 when device 4000 is in this position. 
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Device 4000 uses navigation sensors 4008 and 4010 to manipulate a GUI or 
scroll through an image while viewing display 4002. Navigation sensors 4008 and 
4010 are also used during image capture. This allows the function of one or more of 
the navigation sensors 4008 and 4010 to be shared between display manipulation and 
5 image capture without additional hardware. It also allows capture verification to be 
very intuitive. To verify the image capture, the user would place the bottom side in 
FIG. 4, against a surface and move it around. Alternatively, the user could hold the 
device and move a finger or other body part over at least one of the navigation sensors 
4008 and 4010. The display 4002 would appear to the user to be moving around the 

10 captured image. One of buttons 4004, 4006, 4014, 4016, and 4018 could be used to 
activate a navigate mode that moves a captured image around the display in response 
to the movement of device 4000 or the users finger. Finally, since most GUI and 
scrolling functions do not require the positional accuracy that scanning does, the 
indications of displacement produced by navigation sensors 4008 or 4010 may be 

15 downsampled. This would reduce the compute requirements associated with handling 
the indications of displacement when an image was not being captured. 

Although several specific embodiments of the invention have been described 
and illustrated, the invention is not to be limited to the specific forms or arrangements 
of parts so described and illustrated. The invention is limited only by the claims. 

20 
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